Skip to content

REF: use consistent pattern in tslibs.vectorized #35613

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 7 commits into from
Aug 11, 2020
Merged

REF: use consistent pattern in tslibs.vectorized #35613

merged 7 commits into from
Aug 11, 2020

Conversation

jbrockmendel
Copy link
Member

DO NOT MERGE - Performance takes a hit

We have edited some of the functions in `tslibs.vectorized to do some pre-processing in order to de-duplicate some code. This new pattern is conducive to refactoring out helper functions for further de-duplication.

So far, moving to the new pattern has been performance-neutral or better. But for the functions which this branch moves to the new pattern, performance takes a hit, particularly for dt64arr_to_periodarray. I am at a loss as to why this would be, hoping more eyeballs will help. cc @WillAyd

asv continuous -E virtualenv -f 1.01 master HEAD -b tslibs
[...]
       before           after         ratio
     [d82e5403]       [9f5d9ef6]
     <master~4>       <ref-vectorized>
+      21.4±0.3ms         53.7±6ms     2.51  tslibs.period.TimeDT64ArrToPeriodArr.time_dt64arr_to_periodarr(1000000, 1011, datetime.timezone(datetime.timedelta(seconds=3600)))
+         248±4μs         613±50μs     2.48  tslibs.period.TimeDT64ArrToPeriodArr.time_dt64arr_to_periodarr(10000, 3000, datetime.timezone(datetime.timedelta(seconds=3600)))
+      21.9±0.3ms         53.8±6ms     2.46  tslibs.period.TimeDT64ArrToPeriodArr.time_dt64arr_to_periodarr(1000000, 1000, datetime.timezone(datetime.timedelta(seconds=3600)))
+         238±8μs         574±60μs     2.41  tslibs.period.TimeDT64ArrToPeriodArr.time_dt64arr_to_periodarr(10000, 1011, datetime.timezone(datetime.timedelta(seconds=3600)))
+         268±3μs         617±70μs     2.30  tslibs.period.TimeDT64ArrToPeriodArr.time_dt64arr_to_periodarr(10000, 7000, datetime.timezone(datetime.timedelta(seconds=3600)))
+         246±5μs         562±70μs     2.28  tslibs.period.TimeDT64ArrToPeriodArr.time_dt64arr_to_periodarr(10000, 2011, datetime.timezone(datetime.timedelta(seconds=3600)))
+        282±10μs         636±70μs     2.26  tslibs.period.TimeDT64ArrToPeriodArr.time_dt64arr_to_periodarr(10000, 12000, datetime.timezone(datetime.timedelta(seconds=3600)))
+        274±20μs         616±70μs     2.24  tslibs.period.TimeDT64ArrToPeriodArr.time_dt64arr_to_periodarr(10000, 11000, datetime.timezone(datetime.timedelta(seconds=3600)))
+         270±6μs         599±60μs     2.22  tslibs.period.TimeDT64ArrToPeriodArr.time_dt64arr_to_periodarr(10000, 6000, datetime.timezone(datetime.timedelta(seconds=3600)))
+         248±4μs         544±40μs     2.20  tslibs.period.TimeDT64ArrToPeriodArr.time_dt64arr_to_periodarr(10000, 2000, datetime.timezone(datetime.timedelta(seconds=3600)))
+      23.4±0.4ms         51.3±2ms     2.19  tslibs.period.TimeDT64ArrToPeriodArr.time_dt64arr_to_periodarr(1000000, 2000, datetime.timezone(datetime.timedelta(seconds=3600)))
+         278±8μs         605±70μs     2.18  tslibs.period.TimeDT64ArrToPeriodArr.time_dt64arr_to_periodarr(10000, 8000, datetime.timezone(datetime.timedelta(seconds=3600)))
+         290±9μs        620±100μs     2.14  tslibs.period.TimeDT64ArrToPeriodArr.time_dt64arr_to_periodarr(10000, 9000, datetime.timezone(datetime.timedelta(seconds=3600)))
+        286±40μs         611±60μs     2.14  tslibs.period.TimeDT64ArrToPeriodArr.time_dt64arr_to_periodarr(10000, 10000, datetime.timezone(datetime.timedelta(seconds=3600)))
+        259±20μs         538±60μs     2.08  tslibs.period.TimeDT64ArrToPeriodArr.time_dt64arr_to_periodarr(10000, 1000, datetime.timezone(datetime.timedelta(seconds=3600)))
+       35.5±10ms        73.5±20ms     2.07  tslibs.period.TimeDT64ArrToPeriodArr.time_dt64arr_to_periodarr(1000000, 9000, datetime.timezone(datetime.timedelta(seconds=3600)))
+         300±5μs         610±60μs     2.03  tslibs.period.TimeDT64ArrToPeriodArr.time_dt64arr_to_periodarr(10000, 4006, datetime.timezone(datetime.timedelta(seconds=3600)))
+         289±8μs         586±60μs     2.03  tslibs.period.TimeDT64ArrToPeriodArr.time_dt64arr_to_periodarr(10000, 4000, datetime.timezone(datetime.timedelta(seconds=3600)))
+       38.3±10ms        77.3±20ms     2.02  tslibs.period.TimeDT64ArrToPeriodArr.time_dt64arr_to_periodarr(1000000, 10000, datetime.timezone(datetime.timedelta(seconds=3600)))
+       36.6±10ms        70.9±20ms     1.94  tslibs.period.TimeDT64ArrToPeriodArr.time_dt64arr_to_periodarr(1000000, 11000, datetime.timezone(datetime.timedelta(seconds=3600)))
+        26.1±1ms         49.9±1ms     1.91  tslibs.period.TimeDT64ArrToPeriodArr.time_dt64arr_to_periodarr(1000000, 2011, datetime.timezone(datetime.timedelta(seconds=3600)))
+       38.8±10ms        72.4±20ms     1.87  tslibs.period.TimeDT64ArrToPeriodArr.time_dt64arr_to_periodarr(1000000, 12000, datetime.timezone(datetime.timedelta(seconds=3600)))
+        27.0±3ms       50.2±0.5ms     1.86  tslibs.period.TimeDT64ArrToPeriodArr.time_dt64arr_to_periodarr(1000000, 3000, datetime.timezone(datetime.timedelta(seconds=3600)))
+       35.3±10ms        64.6±10ms     1.83  tslibs.period.TimeDT64ArrToPeriodArr.time_dt64arr_to_periodarr(1000000, 7000, datetime.timezone(datetime.timedelta(seconds=3600)))
+        35.2±9ms        64.3±10ms     1.83  tslibs.period.TimeDT64ArrToPeriodArr.time_dt64arr_to_periodarr(1000000, 6000, datetime.timezone(datetime.timedelta(seconds=3600)))
+        38.0±1ms        67.7±10ms     1.78  tslibs.period.TimeDT64ArrToPeriodArr.time_dt64arr_to_periodarr(1000000, 4006, datetime.timezone(datetime.timedelta(seconds=3600)))
+         358±7μs         637±70μs     1.78  tslibs.period.TimeDT64ArrToPeriodArr.time_dt64arr_to_periodarr(10000, 5000, datetime.timezone(datetime.timedelta(seconds=3600)))
+      9.43±0.2μs         16.7±2μs     1.77  tslibs.resolution.TimeResolution.time_get_resolution('s', 100, tzfile('/usr/share/zoneinfo/Asia/Tokyo'))
+     3.10±0.04μs         5.49±1μs     1.77  tslibs.fields.TimeGetStartEndField.time_get_start_end_field(1, 'start', 'month', None, 5)
+       37.7±10ms        66.5±10ms     1.76  tslibs.period.TimeDT64ArrToPeriodArr.time_dt64arr_to_periodarr(1000000, 8000, datetime.timezone(datetime.timedelta(seconds=3600)))
+        33.5±5ms         57.0±2ms     1.70  tslibs.period.TimeDT64ArrToPeriodArr.time_dt64arr_to_periodarr(1000000, 4000, datetime.timezone(datetime.timedelta(seconds=3600)))
+      26.5±0.1μs        42.8±10μs     1.61  tslibs.tz_convert.TimeTZConvert.time_tz_localize_to_utc(100, datetime.timezone(datetime.timedelta(seconds=3600)))
+        45.3±8ms        70.5±10ms     1.56  tslibs.period.TimeDT64ArrToPeriodArr.time_dt64arr_to_periodarr(1000000, 5000, datetime.timezone(datetime.timedelta(seconds=3600)))
+      24.7±0.2μs        38.2±10μs     1.54  tslibs.tz_convert.TimeTZConvert.time_tz_localize_to_utc(100, <DstTzInfo 'US/Pacific' LMT-1 day, 16:07:00 STD>)
+     1.55±0.08ms       2.36±0.2ms     1.52  tslibs.normalize.Normalize.time_is_date_array_normalized(1000000, datetime.timezone(datetime.timedelta(seconds=3600)))
+         327±4ms        491±100ms     1.50  tslibs.tslib.TimeIntsToPydatetime.time_ints_to_pydatetime('time', 1000000, None)
+      24.7±0.9ms        36.6±10ms     1.48  tslibs.fields.TimeGetDateField.time_get_date_field(1000000, 'dow')
+     3.29±0.06μs         4.87±1μs     1.48  tslibs.fields.TimeGetStartEndField.time_get_start_end_field(1, 'start', 'month', None, 12)
+         338±2ms        497±100ms     1.47  tslibs.tslib.TimeIntsToPydatetime.time_ints_to_pydatetime('time', 1000000, datetime.timezone(datetime.timedelta(seconds=3600)))
+      22.7±0.3μs         32.0±3μs     1.41  tslibs.normalize.Normalize.time_is_date_array_normalized(10000, datetime.timezone(datetime.timedelta(seconds=3600)))
+      12.2±0.5μs         17.1±4μs     1.41  tslibs.period.TimeDT64ArrToPeriodArr.time_dt64arr_to_periodarr(100, 2000, datetime.timezone(datetime.timedelta(seconds=3600)))
+      12.0±0.4μs         16.5±4μs     1.38  tslibs.period.TimeDT64ArrToPeriodArr.time_dt64arr_to_periodarr(100, 2011, datetime.timezone(datetime.timedelta(seconds=3600)))
+      12.1±0.3μs         16.8±2μs     1.38  tslibs.period.TimeDT64ArrToPeriodArr.time_dt64arr_to_periodarr(100, 12000, datetime.timezone(datetime.timedelta(seconds=3600)))
+      12.2±0.2μs         16.6±2μs     1.36  tslibs.period.TimeDT64ArrToPeriodArr.time_dt64arr_to_periodarr(100, 11000, datetime.timezone(datetime.timedelta(seconds=3600)))
+      11.7±0.3μs         15.9±1μs     1.35  tslibs.period.TimeDT64ArrToPeriodArr.time_dt64arr_to_periodarr(100, 3000, datetime.timezone(datetime.timedelta(seconds=3600)))
+      12.6±0.3μs         16.5±2μs     1.31  tslibs.period.TimeDT64ArrToPeriodArr.time_dt64arr_to_periodarr(100, 4000, datetime.timezone(datetime.timedelta(seconds=3600)))
+      8.89±0.6μs         11.3±1μs     1.27  tslibs.period.TimeDT64ArrToPeriodArr.time_dt64arr_to_periodarr(0, 2000, tzfile('/usr/share/zoneinfo/Asia/Tokyo'))
+         115±2μs         143±20μs     1.25  tslibs.normalize.Normalize.time_is_date_array_normalized(10000, <DstTzInfo 'US/Pacific' LMT-1 day, 16:07:00 STD>)
+      12.8±0.3μs         15.9±2μs     1.24  tslibs.period.TimeDT64ArrToPeriodArr.time_dt64arr_to_periodarr(100, 6000, datetime.timezone(datetime.timedelta(seconds=3600)))
+        61.7±2μs         75.0±7μs     1.22  tslibs.normalize.Normalize.time_is_date_array_normalized(10000, tzfile('/usr/share/zoneinfo/Asia/Tokyo'))
+     7.58±0.08ms       8.61±0.1ms     1.14  tslibs.normalize.Normalize.time_is_date_array_normalized(1000000, tzfile('/usr/share/zoneinfo/Asia/Tokyo'))
+     8.80±0.07ms       9.55±0.4ms     1.09  tslibs.normalize.Normalize.time_normalize_i8_timestamps(1000000, tzfile('/usr/share/zoneinfo/Asia/Tokyo'))
-     3.46±0.05μs       3.06±0.1μs     0.89  tslibs.period.TimeDT64ArrToPeriodArr.time_dt64arr_to_periodarr(1, 3000, tzlocal())
-      25.2±0.9ms       21.2±0.5ms     0.84  tslibs.period.TimeDT64ArrToPeriodArr.time_dt64arr_to_periodarr(1000000, 2011, datetime.timezone.utc)
-      2.39±0.1ms       2.01±0.3ms     0.84  tslibs.normalize.Normalize.time_normalize_i8_timestamps(1000000, datetime.timezone(datetime.timedelta(seconds=3600)))
-      47.0±0.5ms       38.7±0.8ms     0.82  tslibs.period.TimeDT64ArrToPeriodArr.time_dt64arr_to_periodarr(1000000, 2011, <DstTzInfo 'US/Pacific' LMT-1 day, 16:07:00 STD>)
-        38.5±3ms       31.2±0.4ms     0.81  tslibs.period.TimeDT64ArrToPeriodArr.time_dt64arr_to_periodarr(1000000, 3000, tzfile('/usr/share/zoneinfo/Asia/Tokyo'))
-        36.4±1ms       29.3±0.7ms     0.81  tslibs.period.TimeDT64ArrToPeriodArr.time_dt64arr_to_periodarr(1000000, 2011, tzfile('/usr/share/zoneinfo/Asia/Tokyo'))
-        25.7±3ms       20.7±0.2ms     0.80  tslibs.period.TimeDT64ArrToPeriodArr.time_dt64arr_to_periodarr(1000000, 3000, datetime.timezone.utc)
-        25.7±3ms       20.5±0.2ms     0.80  tslibs.period.TimeDT64ArrToPeriodArr.time_dt64arr_to_periodarr(1000000, 3000, None)
-     1.88±0.02ms        607±200μs     0.32  tslibs.tz_convert.TimeTZConvert.time_tz_convert_from_utc(1000000, datetime.timezone.utc)

@jreback
Copy link
Contributor

jreback commented Aug 7, 2020

DO NOT MERGE - Performance takes a hit

I think you can do a draft PR to indicate this (of course comments ok too)

@jreback jreback added Refactor Internal refactoring of code Timezones Timezone data dtype labels Aug 7, 2020
@jbrockmendel
Copy link
Member Author

There was a missing type declaration in the periodarray_to_dt64arr func; now this is perf-improving for that func.

       before           after         ratio
     [71a327c4]       [a84c147f]
     <master>         <ref-vectorized>
-     7.74±0.06μs      7.51±0.05μs     0.97  tslibs.period.TimeDT64ArrToPeriodArr.time_dt64arr_to_periodarr(1, 1000, <DstTzInfo 'US/Pacific' LMT-1 day, 16:07:00 STD>)
-         266±5μs        258±0.8μs     0.97  tslibs.period.TimeDT64ArrToPeriodArr.time_dt64arr_to_periodarr(10000, 4006, datetime.timezone(datetime.timedelta(seconds=3600)))
-         221±3μs          214±2μs     0.97  tslibs.period.TimeDT64ArrToPeriodArr.time_dt64arr_to_periodarr(10000, 2011, None)
-     5.25±0.06μs      4.99±0.02μs     0.95  tslibs.period.TimeDT64ArrToPeriodArr.time_dt64arr_to_periodarr(100, 3000, None)
-     5.71±0.03μs      5.42±0.04μs     0.95  tslibs.period.TimeDT64ArrToPeriodArr.time_dt64arr_to_periodarr(100, 10000, None)
-      8.91±0.1μs      8.44±0.05μs     0.95  tslibs.period.TimeDT64ArrToPeriodArr.time_dt64arr_to_periodarr(1, 12000, tzfile('/usr/share/zoneinfo/Asia/Tokyo'))
-     5.43±0.08μs      5.13±0.05μs     0.95  tslibs.period.TimeDT64ArrToPeriodArr.time_dt64arr_to_periodarr(100, 7000, datetime.timezone.utc)
-     5.13±0.06μs      4.84±0.05μs     0.94  tslibs.period.TimeDT64ArrToPeriodArr.time_dt64arr_to_periodarr(100, 1011, datetime.timezone.utc)
-     2.95±0.04μs      2.75±0.06μs     0.93  tslibs.period.TimeDT64ArrToPeriodArr.time_dt64arr_to_periodarr(1, 2000, datetime.timezone.utc)
-      21.2±0.5ms       19.8±0.3ms     0.93  tslibs.period.TimeDT64ArrToPeriodArr.time_dt64arr_to_periodarr(1000000, 1000, datetime.timezone.utc)
-         228±2μs        212±0.6μs     0.93  tslibs.period.TimeDT64ArrToPeriodArr.time_dt64arr_to_periodarr(10000, 2011, datetime.timezone.utc)
-        38.6±3ms       35.8±0.2ms     0.93  tslibs.period.TimeDT64ArrToPeriodArr.time_dt64arr_to_periodarr(1000000, 1000, <DstTzInfo 'US/Pacific' LMT-1 day, 16:07:00 STD>)
-        387±20ns          352±6ns     0.91  tslibs.period.PeriodProperties.time_property('M', 'year')
-      9.69±0.3μs      8.81±0.04μs     0.91  tslibs.period.TimeDT64ArrToPeriodArr.time_dt64arr_to_periodarr(0, 4006, datetime.timezone(datetime.timedelta(seconds=3600)))
-     3.09±0.06μs      2.77±0.03μs     0.90  tslibs.period.TimeDT64ArrToPeriodArr.time_dt64arr_to_periodarr(0, 9000, datetime.timezone.utc)
-        269±10μs          242±1μs     0.90  tslibs.period.TimeDT64ArrToPeriodArr.time_dt64arr_to_periodarr(10000, 9000, datetime.timezone.utc)
-      3.16±0.2μs      2.78±0.04μs     0.88  tslibs.period.TimeDT64ArrToPeriodArr.time_dt64arr_to_periodarr(0, 4006, datetime.timezone.utc)
-      3.41±0.1μs      2.98±0.08μs     0.88  tslibs.period.TimeDT64ArrToPeriodArr.time_dt64arr_to_periodarr(0, 5000, None)
-        25.4±3ms       20.8±0.2ms     0.82  tslibs.period.TimeDT64ArrToPeriodArr.time_dt64arr_to_periodarr(1000000, 2000, None)
-        46.9±8ms      37.4±0.06ms     0.80  tslibs.period.TimeDT64ArrToPeriodArr.time_dt64arr_to_periodarr(1000000, 2011, <DstTzInfo 'US/Pacific' LMT-1 day, 16:07:00 STD>)
-        27.4±5ms       20.7±0.1ms     0.76  tslibs.period.TimeDT64ArrToPeriodArr.time_dt64arr_to_periodarr(1000000, 2011, datetime.timezone.utc)
-        26.7±6ms       19.8±0.2ms     0.74  tslibs.period.TimeDT64ArrToPeriodArr.time_dt64arr_to_periodarr(1000000, 1011, None)
-       44.1±10ms       30.6±0.2ms     0.70  tslibs.period.TimeDT64ArrToPeriodArr.time_dt64arr_to_periodarr(1000000, 7000, tzfile('/usr/share/zoneinfo/Asia/Tokyo'))
-        4.23±1μs      2.81±0.02μs     0.67  tslibs.period.TimePeriodArrToDT64Arr.time_periodarray_to_dt64arr(0, 1011)
-        9.97±3μs      6.38±0.03μs     0.64  tslibs.period.TimePeriodArrToDT64Arr.time_periodarray_to_dt64arr(0, 6000)

Still need to make sure the normalize funcs are neutral or better

@WillAyd
Copy link
Member

WillAyd commented Aug 10, 2020

lgtm save the red CI pieces. Nice cleanup!

@jreback jreback added this to the 1.2 milestone Aug 11, 2020
@jreback jreback merged commit 0639e7f into pandas-dev:master Aug 11, 2020
@jbrockmendel jbrockmendel deleted the ref-vectorized branch August 11, 2020 15:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
Refactor Internal refactoring of code Timezones Timezone data dtype
Projects
None yet
Milestone
1.2
Development

Successfully merging this pull request may close these issues.
3 participants
@jbrockmendel @jreback @WillAyd